The Job Orchestration & Workflow Definition component is responsible for the initial setup, routing, and comprehensive compilation of high-level workflow definitions into executable AWS Step Functions state machine language. This includes handling various step types such as batch, scatter-gather, parallel, sub-pipeline, native, and chooser steps, along with their validation. It acts as the central hub for transforming abstract job specifications into concrete, executable workflows.

Components

Initializer

This component is responsible for the initial setup of a job execution. It reads job data from S3, performs variable substitutions, ensures no recursive launches, copies job data to a designated repository, and writes extended job data for subsequent steps. It acts as the entry point for S3-triggered workflows.

Job Definition Registrar

This component handles the registration and modification of job definitions. It processes incoming requests, potentially edits job specifications, and generates appropriate responses, likely for managing job metadata or configurations within the system.

Job Router

The Job Router component is responsible for determining the correct state machine for execution and generating a unique execution name. It normalizes and shortens filenames to create suitable identifiers for workflow executions.

Chooser Logic

This component implements conditional logic for workflows. It loads values, potentially from S3, and evaluates expressions to make choices or direct workflow paths based on predefined conditions.

Compiler Core

This is the central component for compiling workflow templates into executable state machine definitions. It orchestrates the overall compilation process, including parameter substitution and delegating to other compiler sub-components for handling specific step types and state machine generation.

Compiler State Machine Builder

This component is dedicated to constructing and managing AWS Step Functions state machine definitions. It creates branches, processes individual steps by delegating to specific step handlers, and handles the output of the compiled state machine, including writing it to S3.

Compiler Step Handlers

This component provides specialized logic for processing and generating definitions for various types of workflow steps, including scatter/gather, batch jobs, sub-pipelines, native AWS Lambda steps, parallel executions, and chooser steps. It integrates with utility functions for common operational aspects like logging and retries.

Compiler Validation

This component provides a set of validation functions used by the compiler to ensure that various workflow steps (batch, native, parallel, scatter, subpipe, chooser) adhere to predefined rules and structures, raising errors for invalid configurations.

Compiler Utilities

This component provides common utility functions that support various parts of the compiler and other related components. These utilities include parameter substitution, logging blocks, retry mechanisms, logical name generation, and time string conversions.